feat(rivetkit): trace actor to actor calls - #5732

Stack for rivet-dev/actors

• #5734 docs(rivetkit): add actor tracing docs
• #5733 feat(rivetkit): forward otel sdk warnings to the pino logger
• #5732 feat(rivetkit): trace actor to actor calls 👈
• #5730 feat(rivetkit-core): trace queue sends and receives
• #5731 feat(rivetkit-core): trace raw http requests
• #5729 feat(rivetkit-core): trace scheduled invocations
• #5726 feat(rivetkit): send trace context from clients
• #5723 feat(rivetkit): pass trace context to js and logs
• #5727 feat(rivetkit): trace sqlite operations
• #5721 feat(rivetkit): trace actor invocations
• #5728 fix(pegboard-gateway): forward request ray id to actors
• #5724 test(rivetkit): let driver runtimes take extra environment
• #5722 fix(rivetkit): disable services in driver test runtimes
• main

Get stack: forklift get 5732
Push local edits: forklift submit
Merge when ready: forklift merge 5732

_{change zosqvyzx}

Review

Reviewed the current diff (traces actor-to-actor calls and workflow runs/steps via OpenTelemetry). This is a substantial, well-structured addition. The InvocationType-driven exhaustive matches, the finished-flag idempotency guards on span teardown, and the new sql_efficiency.rs coverage for workflow.kv_get are good signs of care. A few issues worth addressing:

Correctness

1. ActorInvocationTelemetry::trace_context() ignores application_span (rivetkit-rust/packages/rivetkit-core/src/telemetry.rs:671)
   trace_context() only checks self.step_span, but parent_context() gives top precedence to self.application_span. Per the new docs (docs/content/docs/general/tracing.mdx, section "Add application spans"), code that wraps work in tracer.startActiveSpan("my-span", ...) expects c.log/ctx.invocationTraceContext() to report the my-span trace/span ID. Right now SQL calls correctly nest under the application span (via parent_context()), but logs and invocationTraceContext() still report the base invocation span IDs, inconsistent with the span the work actually nests under, and contradicting the example in the new docs.

2. SDK warning can be silently dropped during a sink install/uninstall race (rivetkit-typescript/packages/rivetkit-napi/src/telemetry.rs:64 and :104-124)
   delivered_to_sink (read by the log layer filter) and SdkLogLayer::on_event (the forwarding path) independently read the same SINK: RwLock<Option<Sink>> at two different points in dispatch, with no lock held across both checks. If shutdownTelemetry() / setTelemetryLogSink() flips the sink between these two reads (plausible since it races background export/shutdown warnings emitted by the OTel SDK itself), the log layer can skip printing (assuming the sink will handle it) while the sink layer also skips forwarding (sink now gone), losing the warning entirely. This contradicts the claim in docs-internal/engine/rivetkit-telemetry.md that "each warning is visible once."

Performance

3. Workflow-trace SQLite read gated on tracing::enabled!, not on whether an exporter is actually attached (rivetkit-rust/packages/rivetkit-core/src/actor/context.rs:312)
   start_workflow_span gates an extra _rivet_wf_kv SQLite read behind tracing::enabled!(target: "rivetkit::telemetry", INFO). This is correct today only because rivetkit-napi lib.rs explicitly appends rivetkit::telemetry=off to its log-layer filter. Any other host that installs a plain tracing_subscriber at INFO (a bare rivetkit Rust consumer, a test harness, a future runtime) would make this check return true unconditionally, silently paying the extra SQLite read on every workflow run with no OTLP exporter ever attached, and no diagnostic surfaced. Worth gating on an explicit exporter-configured check instead of subscriber interest.

Code quality (minor)

4. Three Drop impls in telemetry.rs (OutboundCallInvocation ~855, WorkflowStepSpan ~891, SqliteOperationSpan ~914) duplicate the identical "record abandoned" body (otel.status_code = ERROR, error.type = OPERATION_ABANDONED_ERROR_TYPE). Consider a shared record_abandoned(&Span) helper alongside the existing record_outcome.
5. anyhow_error_from_js_reason (rivetkit-typescript/packages/rivetkit-napi/src/actor_factory.rs:987) duplicates the first half of callback_error error-reconstruction logic (parse_bridge_rivet_error(...).unwrap_or_else(...)). A future change to the fallback rule risks being applied to only one copy.
6. takes_message_ray (telemetry.rs:157 / ~590) is a one-off matches!(invocation_type, InvocationType::Workflow) rather than an exhaustive match like the rest of the file InvocationType-driven decisions (record_attributes, otel_kind). A new InvocationType variant that should also adopt an incoming message ray would silently default to false with no compiler signal. (This behavior is called out as intentional in the docs, so it is a mechanism nit, not a correctness bug.)

Test coverage

Test coverage looks solid: tests/driver/actor-telemetry.test.ts covers overlapping-call trace isolation, and sql_efficiency.rs was updated for the new query. It would be worth adding a regression test for finding #1 (application span vs. log / invocationTraceContext() correlation) since it is directly demonstrated in the new docs example.

Conventions

Checked against CLAUDE.md: vbare usage for the new WorkflowTraceContext mirrors the existing RunWakeAt pattern correctly, enum matches are exhaustive with no _ => fallbacks, structured logging fields are used correctly in the two new tracing::warn! calls, and the new parking_lot usages are in forced-sync contexts (NAPI on_event, &self methods) per the Async Rust Locks rule. No violations found there.

🤖 Generated with Claude Code